Parameter Differentiation Based Multilingual Neural Machine Translation

Multilingual neural machine translation (MNMT) aims to translate multiple languages with a single model and has been proved successful thanks effective knowledge transfer among different shared parameters. However, it is still an open question which parameters should be ones need task-specific. Currently, the common practice heuristically design or search language-specific modules, difficult find optimal configuration. In this paper, we propose novel parameter differentiation based method that allows determine language-specific during training. Inspired by cellular differentiation, each in our can dynamically differentiate into more specialized types. We further define criterion as inter-task gradient similarity. Therefore, conflicting gradients are likely language-specific. Extensive experiments on multilingual datasets have demonstrated significantly outperforms various strong baselines sharing configurations. Further analysis reveals configuration obtained correlates well linguistic proximities.